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Abstract 

Networks of neurons in some brain areas are flexible enough to encode new mem- 
ories quickly. Using a standard firing rate model of recurrent networks, we develop a 
theory of flexible memory networks. Our main results characterize networks having the 
maximal number of flexible memory patterns, given a constraint graph on the network's 
connectivity matrix. Modulo a mild topological condition, we find a close connection 
between maximally flexible networks and rank 1 matrices. The topological condition 
is Hi(X;7j) = 0, where X is the clique complex associated to the network's constraint 
graph; this condition is generically satisfied for large random networks that are not overly 
sparse. In order to prove our main results, we develop some matrix-theoretic tools and 
present them in a self-contained section independent of the neuroscience context. 
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1 Introduction 



New memories in some brain areas can be encoded quickly [13J. It is widely believed that 
memories are stored via changes in the synaptic efficacies between neurons. Irrespective of 
the plasticity mechanism, or 'learning rule', used to encode memory patterns, rapid learning 
is perhaps most easily accomplished if new patterns can be learned via only small changes 
in connection strengths between neurons. It may thus be desirable for fast-learning, flexible 
networks to have architectures that enable many memory patterns to be encoded (and un- 
encoded) by only small perturbations of the synaptic connections. Here by 'architecture' we 
mean the pattern of synaptic strengths, or weights, assigned to directed connections between 
neurons. Which network architectures allow maximal flexibility for learning and unlearning 
new memories? 

We study this question in the context of a standard firing rate model of recurrent neural 
networks. Building on the framework of 'permitted' and 'forbidden' sets first introduced in [5] , 
we think of the recurrent network as a gating device that allows only a restricted set of patterns, 
the stored 'memories', to be activated by external feed-forward input. In Theorem 11.21 we 
establish a correspondence between the memory patterns encoded by a recurrent network and 
the set of stable principal submatrices of the network's effective connectivity matrix. We 
then make precise the notion of memory patterns that are 'flexible' in the sense that they 
can be encoded (learned) and unencoded (forgotten) via only small changes to the network 
weights. Our main results, Theorems II .5[ [L71 and [L8| characterize network architectures with 
the maximal number of flexible memories. 

1.1 Network dynamics and architecture 

We consider a standard firing rate model (2j [3] with heterogeneous timescales, 

dx 1 ( n \ 

—j£ = -— x i + ¥ Wi 3 x 3 + ky> for % = 1, n, 

where n is the number of neurons. The real- valued function X{ = X{(t) is the firing rate of the 
ith neuron, is the external input to the ith neuron, and Wij denotes the effective strength 
of the recurrent connection from the jth to the ith neuron. The timescale 7$ gives the rate 
of recovery to rest in the absence of external or recurrent inputs. The nonlinear function 
ip : K — > M>o satisfies <p(y) = whenever y < 0, and ensures that the firing rates Xi, ...,x n 
are non-negative. Although the threshold appears to be zero for all neurons, heterogeneous 
thresholds can easily be incorporated into the 6jS. Note that > 0, while 6, and Wy can take 
on both positive and negative values. The dynamics of the network can be described more 
compactly as 

x = -Dx + (Wx + b), (1) 

where D = diag(ri _1 , T n ~ l ) is the diagonal matrix of inverse time constants, and tp is 
applied to each coordinate when the argument is a vector. Note that we do not require that 
the matrix W respect Dale 's lavfl as the entries are considered to be effective connection 

1 Dale's law states that every element in the same column of the connectivity matrix must have the same sign 
[2]. This is because neurons have either purely excitatory or purely inhibitory synapses onto other neurons. 
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strengths between principal (excitatory) neurons. Negative weights thus reflect effectively 
inhibitory interactions, mediated by the presence of non-specific interneurons that do not 
otherwise enter into the model. We always assume that the diagonal entries of the matrix 
—D + W are strictly negative; otherwise, individual neurons may experience a run-away 
excitatory drive even in the absence of external or recurrent inputs. We will also assume that 
hi is constant in time, though it may vary across neurons. For a given choice of nonlinearity 
<p, the network and its dynamics ([Q) are denoted by the pair of matrices (W, D). 

It is worth noting here that we regard the model JT} as a description of fast-times cale 
dynamics. A more realistic network would also include stochastic fluctuations and adaptive 
mechanisms on a slower timescale [1] , so that 'fixed points' of the fast-timescale dynamics only 
appear for short periods of time. Such fixed points will serve as our model for (transiently) 
activated memory patterns. 

To study network flexibility, we will think of the matrix of effective connection strengths 
W as a sum of two components, 

W = J + A, 

where J corresponds to a fixed and underlying architecture, and A is a matrix of perturbations 
about J. While J reflects broad patterns of connection strengths that may be conserved 
across animals or across time, the matrix A captures individual variations, and is constantly 
changing as a function of the animal's learning and experience. Our main question is then: 

Main Question (version 1). What architectures J allow maximal flexibility for learning and 
unlearning new memory patterns under small perturbations A? 

We will consider this as a question about the strengths, or weights, of the recurrent connections 
between neurons, rather than as a question about which neurons are connected. The pattern 
of allowed connections between neurons will be treated as a constraint. Indeed, networks with 
different strengths of connections, but identical connectivity patterns, may have significantly 
different dynamics and attractors. Moreover, in biological neural circuits the anatomical 
connectivity may be difficult to modify, but the weights of synapses are known to change on 
relatively short timescales in response to learning and experience. 

1.2 Memory patterns as 'stable sets' in threshold-linear networks 

Before addressing our main question about perturbations of network architectures, we inves- 
tigate the set of memory patterns corresponding to any fixed network (W,D). The idea of 
stable fixed points as a model for stored memory patterns in recurrent networks dates back at 
least to [7]. Following the framework of [5], subsets of neurons that are active at stable fixed 
points of ([T]) will serve as our model for stored memory patterns. 

Recall that a fixed point x* is asymptotically stable if there exists an open neighborhood U 
of x* such that lim^oo x(t) = x* for every trajectory x(t) with the initial condition x(0) G U. 
If the fixed point x* has only the property that for all nearby initial conditions x(0) G U the 
trajectory x(t) remains very close to x* for all later times, then x* is a stable point of the 
network dynamics. Note that every asymptotically stable point is stable, but the converse 
is not true. For a given firing rate vector x G R> , we call the subset of active neurons the 
support of x, 

supp(x) = {i | Xi > 0} C {1, n}. 
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Definition 1.1 (stable, marginal, and unstable sets). Let (W,D) be a network on n neurons 
with nonlinearity ip. A non-empty subset of neurons a C {1, ...,n} is a stable set of (W,D) if 
there exists an asymptotically stable fixed point x* of the dynamics (|T|) such that supp(x*) = a, 
for at least one external input vector b G M n . A marginal set of (W,D) is a non-empty subset 
of neurons a for which there exists a stable fixed point of the dynamics (but no asymptotically 
stable fixed point) with support a for at least one external input vector b, and an unstable set 
of (W, D) is a non-empty subset of neurons that is neither stable nor marginal^ 

Stable sets are our model for memory patterns encoded by the network. For a fixed external 
input b there may be one, many, or no asymptotically stable fixed points. As we range over 
all possible inputs, however, we obtain the set of stable sets of the network. Clearly, there 
can be at most 2 n — 1 stable sets in a network of n neurons. In cases of interest, however, the 
recurrent network performs meaningful computations precisely because only a small fraction 
of subsets are stable [5]. Note that a pair of neurons in a stable set need not be connected 
(i.e., we may have Wij = for i, j in a stable set). 

In general, it is very difficult to determine analytically the stable fixed points for a high- 
dimensional, nonlinear dynamical system. If the nonlinearity tp in (pQ) is threshold-linear, 
however, it is possible to use standard tools from linear systems of ordinary differential equa- 
tions to obtain exact results. For this reason, we now restrict ourselves to threshold-linear 
networks, which are networks (W, D) where the nonlinearity is chosen as 



Although sigmoids more closely match experimentally measured input-output curves for neu- 
rons, the above threshold-nonlinearity is often a good approximation when neurons are far 
from saturation [21 H]. If we assume that encoded memory patterns are realized by neu- 
rons firing sufficiently below saturation, it is reasonable to model them as stable sets of the 
threshold-linear dynamics: 



In a (nondegenerate) linear system, x = (—D+W)x+b, there can be at most one fixed point 
of the dynamics for a given input vector b 6 M n ; its stability is characterized by the eigenvalues 
of the matrix —D + W. Unlike linear systems, the threshold-linear network fl2]) can exhibit 
multiple fixed points for the same input vector b. It turns out, however, that stable, unstable 
and marginal sets of neurons in threshold-linear networks have simple characterizations in 
terms of the eigenvalues of the corresponding principal submatrices of — D + W . 

Given an n x n matrix A, and a subset a C {1, ■■■,n}, the principal submatrix A a is the 
square matrix obtained by restricting A to the index set a; i.e., if cr = {si, Sk}, then A a is 
the h x k matrix with {A a )ij = A s . s .. We call a square matrix stable if all its eigenvalues have 
strictly negative real part. We call a matrix unstable if at least one eigenvalue has strictly 
positive real part, and marginally stable if no eigenvalue has strictly positive real part and at 
least one eigenvalue is purely imaginary. Marginally stable matrices are thus on the boundary 
between stable and unstable matrices. 

2 Stable and unstable sets were previously introduced in [5], where they were called "permitted" and "for- 
bidden" sets, respectively. 




X = -Dx + [Wx + b] 



(2) 
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We now state our characterization of stable sets in terms of the stability of principal 
submatrices: 

Theorem 1.2. Let (W,D) be a threshold-linear network. A subset of neurons a is a stable 
set of (W, D) if and only if the principal submatrix (-D + W) a is stable. Similarly, a is a 
marginal set or an unstable set of (W, D) if and only if (-D + W) a is marginally stable or 
unstable, respectively. 

In the case of symmetric threshold-linear networks, where the matrix W is symmetric, the 
equivalence between stable ('permitted') sets and stable principal submatrices was shown in 
[5] . We give the proof of Theorem 11.21 in Section |2j 

1.3 G-constrained networks 

There are two ways in which a zero-weight connection between two neurons may arise. On 
the one hand, there may be a lack of anatomical connectivity between the neurons. On the 
other hand, many synaptic connections that appear anatomically are not functional - these 
are referred to as silent synapses [10]. While the first type of zero- weight connection cannot 
be perturbed without major changes to the network architecture, silent synapses may become 
active via small modifications. In addressing our main question, we are therefore interested 
in characterizing maximally flexible networks where some connections are constrained to be 
zero, while the remaining weights (some of which may also be zero) can be modified by small 
perturbations of the network. The following definitions hold for general networks, not just 
threshold-linear ones. 

Let G = (V, E) be a simple graph with vertices V = {1, n} and edges E. We say that an 
n x n architecture matrix J is constrained by the graph G if Jy = for all edges (ij) ^ E. By 
abuse of notation, we often use G to refer to the edge set E. Note that all architectures on n 
neurons are constrained by the complete graph G = K n . If for (ij) G G the entry J±j = 0, we 
say that there is a silent connection from neuron j to neuron i. This mirrors the phenomenon 
of silent synapses in the brain. 

We define an e -perturbation of a network architecture J to be a matrix A whose entries all 
satisfy [A^] < e. We say that an e-perturbation is consistent with G if the matrix A satisfies 
Aij = for all (ij) ^ G. In other words, consistent e-perturbations can only perturb entries 
that are not constrained to be zero (including silent connections). 

When considering an architecture J that is constrained by a graph G, we refer to the 
network as (J, D)g- For a given ip, we use the following notation for the set of all G-constrained 
network architectures: 

M(G) := {(J,D) G } = {(J,D) | Jy = for all (ij) $ G}. 

Note that the set of constrained architectures is independent of the nonlinearity tp. When 
G = K n is the complete graph (no constraint), we will simply write M(n) := M(K n ). If 
G\ C G2, then Af(Gi) C M(G-2). An e-perturbation of a network (J,D)g will always be 
assumed to be consistent with G, and hence to stay within Af(G). 
We can now state our main question a bit more precisely: 
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Main Question (version 2). For a given constraint graph G, what network architectures 
(J, D)g G -A/"(£?) allow the maximal number of subsets of neurons that can become both 
stable sets (learned/encoded) and unstable sets (forgotten/unencoded) via arbitrarily small 
e-perturbations of J? 

We call such subsets of neurons flexible memory patterns. 

1.4 Flexible memory patterns as 'flexible cliques' 

Intuitively, a flexible memory pattern is a subset of neurons that can become both a stable set 
and an unstable set via only small modifications of the network's connection strengths. Ideally, 
these modifications should be specific enough not to change the stability of any other subsets. 
Moreover, we would like flexible memory patterns to correspond to subsets of neurons that 
are unconstrained in their connections to each other. In other words, these subsets of neurons 
should be all-to-all connected in the sense that all mutual connections can be perturbed, 
although some may be zero-weight (silent) connections. We model such memory patterns as 
'flexible cliques'; a precise definition is given below. 

Recall that a clique in a graph G is a subset of vertices that are all-to-all connected, and the 
clique complex of G, denoted X(G), is the set of all cliques. We will say that a C {1, ...,n} 
is a stable clique of the network (W,D)g if ct is a stable set and o G X(G). Similarly, an 
unstable clique is an unstable set a such that a G X(G), and a marginal clique is a marginal 
set a such that a G X[G\ Because the stability of a matrix forces one or more of its 
principal submatrices to be stable (see Lemma [3 .151) . one cannot require that a perturbation 
that changes the stability of a marginal clique in a threshold-linear network also preserves all 
other marginal cliques. For this reason we introduce the notions of 'maximally stable' and 
'minimally unstable' cliques. A maximally stable clique is a stable clique that is not properly 
contained in any larger stable clique; a minimally unstable clique is an unstable clique that 
does not properly contain any other unstable clique. We can now define flexible cliques: 

Definition 1.3 (flexible clique). We call a subset of neurons a C {l,...,n} a flexible clique 
of a network architecture on n neurons, (J, D)g, if for every e > there exist e-perturbations 
A s and A u , consistent with G, such that a is a maximally stable clique of (J + A s , D)a and 
a minimally unstable clique of (J + A u , D)q. 

Flexible cliques are our model for flexible memory patterns. All flexible cliques are marginal 
cliques, but the converse is not true (see Section 14.11) . This is because the flexibility of a 
marginal clique depends on the relationship of this clique to other cliques in the network. In 
general, it is difficult to determine whether or not a marginal clique is flexible in a network 
with many marginal cliques. We are interested in precisely this case, as we look for properties 
of networks with the maximal number of flexible cliques. 

1.5 Statement of the Main Results 

Our main results all concern threshold-linear networks only. Consequently, from now on we 
assume M{n) and M{G) correspond to sets of unconstrained and G-constrained threshold- 
linear networks, respectively. First, we define what we mean by the 'flexibility' and 'rank' of 
a network: 
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Definition 1.4 (network flexibility, rank, and completion). We define the flexibility of a 
network as the number of flexible cliques, and denote it: flex( J, D)q. We define the rank of a 
network (J, D)q to be the rank of the matrix — D + J. We say that a G-constrained network 
on n neurons, (J, D)g, has a rank k completion if there exists a network (J, D) G Af(n) of 
rank fc such that D = D and = Jy for all i = j and all distinct pairs (ij) G G. 

We now further refine our main question: 

Main Question (version 3). For a given constraint graph G, what threshold-linear networks 
(J, D)g G A/"(G) attain maximum flexibility? 

Note that the flexibility of a G-constrained network (J, D)g is bounded by the total number 
of non-empty cliques in the corresponding clique complex X(G), and by the fact that single 
neurons can not be flexible cliques because —D + J has strictly negative diagonal. Thus, 

&ex(J,D) G <\X(G)\-n-l, 

where n is the number of neurons. In A/"(n), the flexibility can be at most 2 n — n — 1. Most 
networks, however, have no flexible cliques. 

The rank of any network (J,D)g is at least 1. For threshold-linear networks, rank 1 
networks are good candidates for attaining maximum flexibility because all but the lxl 
principal submatrices are marginally stable. Indeed, we find that rank 1 networks attain 
the upper bound on flexibility in Af(n), and a similar statement is true about G-constrained 
networks: 

Theorem 1.5. All rank 1 threshold-linear networks on n neurons are maximally flexible in 
N(n), and have flexibility 2 n — n — 1. All G-constrained threshold-linear networks with a rank 
1 completion are maximally flexible in N(G), and have flexibility \X(G) \ — n — 1. 

The proof is given in Section 14.21 

Can any networks other than ones that are rank 1, or have rank 1 completions, attain 
maximal flexibility? The following example demonstrates that a G-constrained network can 
be maximally flexible without having a rank 1 completion. 

Example 1.6. Let G = (V,E) be a simple graph with vertices V = {1,2,3,4} and edges 
E = {(12), (23), (34), (41)}. G is a cycle on 4 vertices, so the clique complex X{G) has no 
cliques of size greater than 2. Consider the threshold-linear network (J, D)g G jV(G), where 

1 \ 

1 

-1 1 

1 -1/ 

Using Lemma [4.11 (see Section H~T1) . it is easy to see that all a G X(G) such that \a\ = 2 are 
flexible cliques, and hence (J,D)g is maximally flexible in A/"(G). Despite this, (J,D)g does 
not have a rank 1 completion, since there is no rank 1 matrix that agrees with — D + J on all 
of its nonzero entries (cf. Example 13.81 in Section l3~2l) . 



-D + J 



( -1 2 

1/2 -1 

1 

\ i o 
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It turns out, however, that examples of this kind can be eliminated by imposing a simple 
topological condition on the clique complex of the constraint graph G. Note that a clique 
complex is an abstract simplicial complex, whose homology groups can be computed using 
simplicial homology. The following is our main result: 

Theorem 1.7. Let (J,D)g be a maximally flexible threshold-linear network in Af(G), and 
suppose that the clique complex X(G) satisfies Hi(X(G);Z) = 0. Then (J,D)g has a rank 1 
completion. In particular, (J, D)g has no silent connections. 

The vanishing of Hi(X(G); Z) may at first appear to be a strong condition, but in fact it is 
generically satisfied for large random networks that are not overly sparse. For example, it was 
recently shown in [9] that if G is an Erdos-Renyi graph with edge probability p (i.e., a random 
graph on n vertices with independent connection probability p between any pair of vertices), 
then p > n~ a with a < 1/3 implies that the probability of Hi(X(G); Z) = approaches 1 as 
n — > oo. For n = 10 4 neurons, the first homology group of the clique complex is expected to 
vanish for connection probabilities as low as p = .05. 

The proof of Theorem ll.7l is given in Section l4~3l For the complete graph K n , H\(X (K n ) , Z) = 
0, thus the following result for unconstrained networks is a corollary of Theorems 11.51 and ll.7t 

Theorem 1.8. A threshold-linear network is maximally flexible in M{n) if and only if it is a 
rank 1 network. 

We give in Section 14.31 a separate proof of this theorem independent of homology / cohomology 
arguments. 

1.6 Discussion 

In this article we have laid out the foundations for a theory of flexible memory networks - 
that is, for recurrent networks with memory patterns that can be both encoded (learned) and 
unencoded (forgotten) by arbitrarily small perturbations of the matrix of connection strengths 
between neurons. Given a constraint graph G of allowed connections, we have found, modulo a 
mild topological condition, that maximally flexible networks in M{G) correspond precisely to 
networks (J, D)g that have a rank 1 completion. These results may provide valuable insights 
for understanding fast-learning, flexible networks in the brain. 

Our results are based on an analysis of the fixed point attractors of a standard firing rate 
model fl2]). We emphasize, however, that we regard this only as a model of the fast-times cale 
dynamics of a recurrent network; in a more comprehensive model, additional elements such as 
stochastic fluctuations, changing external inputs, and adaptive variables on a slower timescale 
will all lead to frequent transitions between attractors of the fast-timescale equations (T5]). This 
approach has proven particularly fruitful in modeling of hippocampal networks, where "bump 
attractor" models on a fast timescale form integral building blocks to more comprehensive 
models that have been successful in describing experimental observations from simultaneously 
recorded populations of neurons [TU [TTJ [121 IS]- 

Thus far we have only considered the extreme case of maximally flexible networks. For 
these networks, arbitrarily small perturbations of the synaptic weights between neurons are 
sufficient to encode new memory patterns. Nevertheless, larger perturbations will be necessary 
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for these patterns to be robust in the presence of various plasticity mechanisms that are 
engaged during ongoing spontaneous activity. For any given learning rule and/or constraint 
on the rate of synaptic changes, however, maximally flexible networks have the best chance 
to quickly encode (or unencode) new memories as stable fixed points of the dynamics. 

It may be possible to extend these results to other flexible networks. Do all rank k 
networks have the same flexibility? What is the general relationship between the flexibility of 
a network and its rank? What is the appropriate generalization of the topological condition 
in Theorem 11.71 when considering higher rank completions? We leave these questions to future 
work. 

The remainder of this paper is organized as follows. In Section [2] we prove Theorem II. 2[ 
making the connection between fixed points of the recurrent network dynamics and the sta- 
bility of principal submatrices. In Section [3] we develop some matrix-theoretic results that 
are critical to proving our main theorems. This section is self-contained, independent of the 
context of neural networks. In Section H] we prove our main results, Theorems II. 5[ 11.71 and 

m 

2 Stable sets correspond to stable principal submatrices 

Recall that a threshold-linear network (W, D) has dynamics described by the system, 

± = -Dx + [Wx + b} + , (3) 

with x G 1R> the firing rate vector, D a diagonal matrix with strictly positive diagonal entries, 
and —D + W annxn matrix having strictly negative diagonal entries. For such networks, one 
is able to obtain qualitative characterizations (stable, marginal, unstable) of sets of neurons. 
In what follows, we consider fixed points of fl3]) for a fixed input vector b G M n . 

Suppose there exists a fixed point of ()3]) with all neurons firing, i.e. x* > 0. Since Dx* > 0, 
we can drop the threshold in a small neighborhood of this fixed point, where the system is 
described by the linear system x = (-D + W)x + b. If the matrix —D + W is invertible, the 
linear system has exactly one fixed point, (D — W)~ 1 b, although this fixed point may or may 
not be in the regime M> where all neurons are firing. Either x* = (D — W)~ 1 b > 0, or we 
have a contradiction and there is no x*. As is well-known for linear systems, the fixed point 
x* is asymptotically stable if and only if the matrix — D + W is a stable matrix. 

In addition to a possible fixed point with all neurons firing, the system ([3]) may also 
have fixed points corresponding to proper subsets of neurons with non-zero firing rate. Let 
a = supp(x*) C {1, ...,n} be the subset of neurons that are firing at the fixed point x*, with 
the complement a representing the remaining (silent) neurons. To describe these types of 
fixed points, we reorder the neurons and write 

x= (x!)' and b= {l)- 

The system ([3]) becomes, 

Xa = -D 5 X & + [W & aXa + W &tT X a + + , 
X a = -D a X a + [WaaXv + W aa X a + b a ] + , 



W 



w, 



(7(7 
(7(7 



w, 



(7(7 
(7(7 
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and, since x* 5 = 0, the fixed point equations for x* simplify to: 

= \W 9tr xl + be]+, 
D a x* a = [W aa x: + b a ] + . 

Since D a x* a > 0, we can drop the threshold in the second equation and obtain 

{Da ~ W aa )x% = K. (4) 

However, a solution x* of this equation only yields a valid fixed point if x* > and W arT x a + 
h < 0. 

To analyze the stability of a fixed point x* with supp(x*) = a, we make the following 
change of coordinates. Let ( ^ j = x — x*, with y E K>q and z E R>o- Then x* is a stable 
fixed point of if and only if the origin is a stable fixed point of: 

y = -D a y + [W a ay + W aa z + (W aa xl + ba))+, 

z = -D a {z + x*) + \W <T9 y + W aff z + (W m x*+b a )] + . 

The existence of the fixed point (y = 0, z = 0) implies that W a(T x*+b a < and W aa x* a -\-b a > 0. 
If we further assume that Wa a x* a + b& < 0, then there exists an open neighborhood of the 
origin for which the sign of each of the thresholded terms is determined by the constant terms 
(those that do not involve y or z). In this neighborhood, we can simplify the thresholds and, 
using (j3J), the equations take the form, 

V = -D„y 

z = -D a z + W„ 9 y + W aa z. 

Because the system is exactly linear in a neighborhood of the fixed point, x* is asymptotically 
stable if and only if the matrix 



M 



-D 9 

W a9 -D a + W a 



is stable. Similarly, x* is stable but not asymptotically stable if and only if M is marginally 
stable, and x* is an unstable fixed point if and only if M is unstable. Finally, note that the 
stability of M is equivalent to the stability of —D a + W aa = (-D + W) a . 

We collect these observations into the following characterization of fixed points in threshold- 
linear networks: 

Proposition 2.1. Consider the system ()3]) ; for a threshold-linear network (D, W) on n neu- 
rons with fixed input b, and let a C {1, n} be a subset of neurons. The following statements 
hold: 

(i) A point x* with supp(x*) = a is a fixed point if and only if x* satisfies: 

(a) (D a - W aa )xl = b a , 

(b) x* a > 0, and 
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(c) b- a < -W 9a x*. 



In particular, ifdet(D a — W aa ) ^ 0, then there exists at most one fixed point with support 
a, and it is given by x* = (D a — W^) -1 ^. 

(ii) Supposex* is a fixed point with supp(x*) = a. Ifb„ < —Wa a x%, thenx* is asymptotically 
stable if and only if the principal submatrix (-D + W) a is stable. Similarly, x* is stable 
but not asymptotically stable if and only if (-D + W) a is marginally stable, and x* is 
an unstable fixed point if and only if (-D + W) a is unstable. 

Using this Proposition, we can now prove Theorem 11.21 

Proof of Theorem 11.21 We begin with the first statement. (=^>) Let a be a stable set of 
(W,D), and choose b such that there exists an asymptotically stable fixed point x* of 
with supp(x*) = a. By part (i) of Proposition 12.11 it is clear that we can choose b such that 
ba < —Wv a x*. It then follows from part (ii) that (-D + W) a is stable. 

(<=) Now suppose that (— D+W) a is stable. We construct b, the input vector in (j3J), so that 
the corresponding fixed point with support a is asymptotically stable. Let b a = (D — W) a l a , 
where l a is the vector of all ones. Letting x* be the firing rate vector with supp(x*) = a and 
x* a = l a > 0, we choose b 5 to satisfy b^ < -W^ a x* a . Note that (D a - W aa )x* a = b a . For this 
choice of b, it thus follows from part (i) of Proposition 12.11 that x* is a fixed point, and by part 
(ii) that x* is asymptotically stable. Hence, a is a stable set of (W, D). 

Similar arguments using Proposition 12.11 can be used to show that a is a marginal or 
unstable set of (W, D) if and only if (— D + W) a is marginally stable or unstable, respectively. 

□ 

As a result of Theorem II .2\ we see that in order to investigate stable, unstable, or marginal 
sets of neurons in threshold-linear networks we need to understand the stability of principal 
submatrices. 



3 Matrix-theoretic results 

In this section, we prove results concerning real matrices with strictly negative entries on 
the diagonal. These results will be critical for Section HJ where we prove our main theorems 
regarding maximally flexible networks. This section is self-contained, however, and the results 
can be understood independently of the context of neural networks. 

Throughout this section, A is an n x n matrix with real coefficients and strictly negative 
entries on the diagonal. The matrix Sa = (%) is the sign matrix associated to A; this is a 
matrix whose entries e^- G {±1,0} are the signs of the corresponding entries of A. To the 
matrix A we also associate the graph Ga, which we call the connectivity graph of A. It is the 
simple graph that includes each edge (ij) unless A^ = A^ = 0. Let X{Ga) be the clique 
complex associated to the graph Ga'-, we call this the clique complex associated to the matrix 
A. Note that a clique complex is an abstract simplicial complex. For any simplicial complex 
X and abelian group Q, we denote the associated simplicial homology and cohomology groups 
as Hi(X;Q) and iiP(X;(?), respectively. Finally, mirroring Definition II. 4[ we call an n x n 
matrix A a completion of A if A\j = Ay for all i = j and all distinct pairs (ij) G Ga- 
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3.1 Bipartite matrices 

Bipartite matrices play an important role in Section 13.21 

Definition 3.1 (bipartite matrix). We say that a real- valued, n x n matrix A is a bipartite 
matrix if the index set {1, n} can be partitioned into two disjoint sets, a and a, such that: 

1. if i G u and j G a , both Aij > and Aji > 0. 

2. if i, j G a or i, j G a, both Ay < and Aji < 0. 

This definition is equivalent to the condition that there exists a permutation of the indices 
{1, ...,n} such that the sign pattern of A takes on the block-form: 



sgn(A) = 




where "+" indicates a submatrix with all nonnegative entries, and "— " a submatrix with 
all nonpositive entries. From this observation it is easy to see that all rank 1 matrices with 
negative diagonal are bipartite: 

Lemma 3.2. Let A be a real n x n matrix with rank A = 1 and An < for all i — 1, ...,n. 
Then there exists a permutation of the indices such that the sign pattern of A is of the form ([5]) . 
In particular, A is a bipartite matrix. 

The following result gives a sufficient condition for bipartiteness of a matrix in terms of the 
associated clique complex. 

Lemma 3.3 (bipartite lemma). Let A be a real-valued n x n matrix with strictly negative 
diagonal, sign matrix £ a = (^ij), connectivity graph Ga, and clique complex X (G a) ■ Suppose 
that 

(i) tijtji = 1, whenever (ij) G X{Ga), 
(ii) eij€jk€ki = —1, whenever (ijk) G X{Ga), and 
(in) H 1 (X(G A );Z 2 ) =0. 
Then A is a bipartite matrix. 

Proof. It is convenient to think of Z 2 = {±1}, the multiplicative group with two elements. 
Consider the co-chain complex, with X = X(Ga), 

C°(X; Z 2 ) A C\X- Z 2 ) ^ C 2 (X; Z 2 ) ^ ■ ■ ■ ^4 C n (X; Z 2 ) ^ 0. (6) 

The maps are the usual coboundary operators. For example, ^({^j}) — { e «j}; where Cij — 
= ViVj. Similarly, ^({e^}) = {f ijk }, where f ijk = e^e^ey = e i:j e jk e k i. 
By (i) we have {%} G C X (X; Z 2 ), while (ii) implies that {— e^} G Ker 5\ (we need the minus 
sign because kernel elements map to +1). Using (iii), we conclude that {— e^} G Im<5 . Hence, 
there exists a vertex labeling {z/j} G C°(X;7j 2 ) such that — eij = ViVj whenever (ij) G X. Let 
a = {i | Vi = +1} and a — {% \ Vi — —1}, with a U o = {l,...,n}. The sign of an edge, 
€ij = —ViVj, can only be positive if i G a and j G o or if i G a and j G a, and can only be 
negative if i,j G cr or z, j G o". This proves that the matrix A is bipartite. □ 
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Example 3.4. To see why we need the cohomology condition in cases where there are zero 
entries, consider the matrix 





-1 


1 








1 > 




1 


-1 


1 













1 


-1 


1 













1 


-1 


1 


I 


1 








1 


-1 J 



The graph Ga is a cycle on 5 vertices, and A is not a bipartite matrix. Nevertheless, A satisfies 
conditions (i) and (ii). 

When the matrix A has no zero entries, however, this kind of example cannot occur. The 
clique complex of the complete graph, X(K n ), is contractible, so H l (X(K n ); Z2) = and we 
obtain the following corollary: 

Corollary 3.5. Let A be a real-valued, n x n matrix with strictly negative diagonal and sign 
matrix Sa = (tij) with all entries nonzero. Suppose that (i) tijtji = 1, for all (ij), and (ii) 
€ij€jk£ki = —1; f or oil distinct triples (ijk). Then A is a bipartite matrix. 

3.2 Vanishing principal minors and rank 

Recall that, given an n x n matrix A and a subset a C {l,...,n}, the principal submatrix 
A a is the square matrix obtained by restricting A to the index set a. The determinant of a 
principal submatrix is called a principal minor. 

It is well-known that if the rank of A is A;, then all principal minors larger than k x k 
must vanish. The converse is not true. For example, consider a strictly upper-triangular n x n 
matrix. It can have rank up to n — 1, yet each and every principal minor vanishes. In the case 
of matrices with strictly negative diagonal entries, however, we do have a kind of converse; 
this is the subject of Lemma 13.71 and Proposition 13. 9[ the main result in this section. We 
begin with a simple technical lemma that will be used throughout. 

Lemma 3.6. Let A be a real-valued n x n matrix with strictly negative diagonal entries and 
clique complex X(Ga)- Suppose that all 2 x 2 and 3x3 principal minors corresponding to 
cliques in X(Ga) vanish. Then the matrix A has symmetric sign matrix Sa, and its entries 
satisfy 

AuAjj = AijAji, for (ij) G X{G A ), and (7) 
AuAjjAkk = AjAjkAki, for (ijk) G X(G A ). (8) 

Proof. The first set of relations (J7]) is obvious, and ensures that Sa is symmetric and A^ ^ 
for all (ij) G X(G A )- This, together with the vanishing of 3 x 3 principal minors, yields: 

2A ii A jj A kk = AijAjkAki + A kj AjiA ik , for (ijk) G X(G A )- 

Given a triple (ijk) G X(G A ), denote x = AuAjj Akk and y = AijAj^Au, and note that 
1/ / 0. Using again (JTJ), we can write A k jAjiA ik = x 2 /y. The above 3x3 condition becomes 
2xy = y 2 + x 2 , and we conclude that x = y, which yields OH])- □ 
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We can now show that the vanishing of all 2 x 2 and 3x3 principal minors suffices to 
guarantee that A has rank 1. 

Lemma 3.7 (rank 1). Let A be a real-valued, n x n matrix with strictly negative diagonal 
entries such that all 2 x 2 and 3x3 principal minors vanish. Then Ga is the complete graph, 
A is a bipartite matrix, and rank A = 1 . 

Proof. Since all 2 x 2 principal minors vanish, it follows that for each i ^ j we have AijAji = 
AaAjj 7^ 0, and Ga is therefore the complete graph. By Lemma I3.6[ the relations ([7]) and ([8]) 
are satisfied for all pairs and triples of distinct indices. Anchoring ourselves on the first row 
and column, we find that any entry of the matrix can be written as: 

. _ AnAjjAii _ AqAij _ AnAij 
13 ~ A n A u ~ A u ~ A n 

Let u = (An, A 2 i, A n i) T be the first column vector of A and v = (A n , A u , Ai n ) the first 
row vector. Then A = (An)~ l uv, which is manifestly rank 1. It follows from Lemma T3.2I that 
A is bipartite. □ 

Can we generalize this result for matrices with zeroes - i.e., for matrices A such that Ga 
is not the complete graph? Here we are looking for conditions that ensure the matrix A has 
a rank 1 completion, where the entries with zeroes are treated as "unknown" entries that 
can be completed to any value. In this case, we can require only that all 2 x 2 and 3x3 
principal minors in the clique complex X(Ga) vanish. The following example shows that such 
a requirement is insufficient to guarantee the existence of a rank 1 completion. 

Example 3.8. Consider the matrix 



A = 



I 


-1 


a 





1/d \ 




1/a 


-1 


b 










1/6 


-1 


c 


V 


d 





l/c 


- 1 / 



This matrix has G A = (V,E), where V = {1,2,3,4} and E = {(12), (23), (34), (41)}. G A is 
a cycle on 4 vertices, and the clique complex X(Ga) = Ga since there are no 2-dimensional 
faces. Note that all 2 x 2 principal minors corresponding to 2-cliques in X(Ga) vanish, and 
there are no 3 x 3 ones to check. Does this matrix have a rank 1 completion? Generically, 
the answer is "No." In fact, it is easy to see that a rank 1 completion exists if and only if 
abed = 1. 

The intuition we gain from this example is that there is a topological obstruction to a 
matrix having a rank 1 completion. It is the presence of a closed but hollow cycle in X(Ga) 
that prevents A from having a rank 1 completion. In fact, if we added or removed an edge 
from the graph Ga in Example 13.8} we would have a rank 1 completion without any further 
condition other than the vanishing of 2 x 2 and 3x3 principal minors det A a for a G X[Ga\ 
The following Proposition gives topological conditions that guarantee the existence of a rank 
1 completion. Note that a condition ensuring that A is bipartite is needed to show that A (as 
opposed to only \A\) has a rank 1 completion. 
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Proposition 3.9. Let A be a real-valued nxn matrix with strictly negative diagonal and clique 
complex X (G a) ■ Let \A\ denote the matrix of absolute values of A. Suppose that detA a = 
for all o G X(Ga) such that \o~\ =2 or 3. Then, 

(a) H 1 (X(Ga)\^2) — ==>- A is a bipartite matrix. 

(b) H 1 (X(Ga)',7$.) = ==> \A\ has a rank 1 completion. 

(c) H 1 (X(G A );R) = iJ 1 (X(G A );Z 2 ) = A has a rank 1 completion. 

Proof. Let X = X(Ga)- First, observe that we satisfy the conditions of Lemma [3.61 and so 
we have relations (J7J and flS}. 

(a) : Let Sa = ( e ij), with G {±1,0}, be the sign matrix of A. Relations (JTj) and (jHJ) imply 

€ij€ji = 1, for (ij) G X, and 
tijtjk^ki = -1, for (ijk) G X. 

Since we also have H 1 (X; Z2) = 0, it follows from Lemma [3.31 that A is a bipartite matrix. 

(b) : For every theft is non-zero, introduce the following (real) variables: 



Lij : = In 



I A? I 



\[A^Aj 



33 



In these variables, the relations (IT]) and (jHJ) are equivalent to antisymmetry and cocycle con- 
ditions on the Lif 

Lij + Lji = 0, for (ij) G X, and (9) 
L - + L jk + L ki = 0, for (ijk) G X (10) 

Now consider the co-chain complex 

C°(X; R) A C X (X; R) ^> C 2 (X; R) A • • - ^4 C n (X; R) ^> 0. (11) 

where C k {X;R) is the group of fc-cochains with coefficients in R. C°(X;R) corresponds to 
vertex-labelings, C 1 (X; R) is the set of edge-labelings, etc. As usual, the coboundary operators 
are 4({./', : ,,. }) = {& ,...,i fe+ i}> where 



9io,—,ih+i ^ f; 

3=0 



k+1 

'-IV'/. 



and o 5 fc = 0. By assumption, H l (X\ R) = 0, so Im5 = Ker 

Let L = (L^), for (17) G X Observe that © implies that L G C^XjR), while the 
cocycle condition (TlOl) implies that L G Ker^i. It follows that L G Im<5o, so there exists an 
a G R™ = C°(X; R) such that Lij = aj — a^. This implies that, for each Aij 7^ 0, 



\Aij\ = y /A ii A jj e Li f = \/\A~\e a y\Ajj\e^. 
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Let u,v G W 1 with Ui = v/jA^]e ai and Vj = \ Ajj | e aj . Since |A| and uv T agree on on all 
nonzero entries of the matrix \A\— uv T is a rank 1 completion of \A\. 
(c): Recall from the proof of part (a) that is the sign of Ay, so that A^ = e^Ay] for each 
entry of A. Following the proof of Lemma |3~3| there exists a vertex labeling G C°(X; Z2), 
with z/j G {±1}, such that = —Vifj whenever (ij) G X. Choose u,v G M n as in the proof 
of part (b), so that \A^\ = UiVj whenever (ij) G X. Now consider u, v G M n where «j = — i^Wj 
and = ^jfj. It follows that Ajj = Ujtij whenever Ay 7^ 0. The matrix A = uv T is thus a 
rank 1 completion of A. □ 

Remark 3.10. Note that Lemma I3TT1 follows easily from Proposition 13.91 since the clique com- 
plex of the complete graph X(K n ) is contractible, so the conditions H l (X(K n ); Z 2 ) = and 
^(X^Kn); K) = are trivially satisfied. In Theorem II .7\ for simplicity we use instead the 
somewhat stronger condition H 1 (X(G);Z) = 0. If #i(X(G);Z) = 0, then H l (X(G);Z 2 ) = 
H 1 (X(G);W) = 0; this follows from the following well-known observation. 

Lemma 3.11. Let X be a simplicial complex. Assume that Hi(X; Z) = 0. Then H 1 (X, Q) = 
0, for every abelian group Q . 

Proof. This is a consequence of the Universal Coefficients Theorem [6], which for an abelian 
group Q yields the short exact sequence 

— ► Ext (X, Z), Q) — > H q (X, Q) Rom(H q (X, Z),g)^0 

for all q > 1. Note that for H a free abelian group, Ext(H, Q) = 0. Since H (X, Z) is always 
free, the above for q = 1 yields fl^pf, 0) Hom(i/ 1 (X, Z),Q) = 0. □ 

3.3 Stable and marginally stable matrices 

Recall that all flexible cliques are marginal cliques, and by Theorem 11.21 the marginal cliques 
correspond to marginally stable principal submatrices. Therefore, to make the connection 
to flexible cliques in Section HJ we need to consider what happens to a matrix when its 
principal submatrices are marginally stable, which is not quite the same as having vanishing 
determinant. 

Recall that a matrix is marginally stable if no eigenvalue has strictly positive real part 
and at least one eigenvalue is purely imaginary. In the case of symmetric matrices, marginal 
stability implies the existence of a zero eigenvalue, and hence vanishing determinant. This is 
not in general true for non-symmetric matrices. However, 2x2 and 3x3 marginally stable 
matrices with negative diagonal entries do have the following characterization: 

Lemma 3.12. (i) Let A be a 2x2 real matrix with strictly negative diagonal entries. Assume 
that A is marginally stable. Then det(A) = 0, and the sign matrix £a is a symmetric matrix 
with all entries nonzero. 

(ii) Let A be a 3 x 3 real matrix with strictly negative diagonal entries. Assume that A is 
marginally stable. Then either det(A) = 0, or det(A) ^ and A has a 2 x 2 stable principal 
submatrix. 
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Proof, (i) The matrix A must have a purely imaginary eigenvalue. Since tr(A) < 0, this 
eigenvalue must be 0, and thus det(A) = 0. As a consequence, the off-diagonal entries of the 
matrix A must have the same sign and are both nonzero. 

(ii) Let Ai be a purely imaginary eigenvalue of A. We have two possibilities: either Ai = and 
thus det(A) = 0, or Ai ^ 0. In the second case, the conjugate Ai = — Ai is also an eigenvalue, 
and since ti{A) < 0, the third eigenvalue A3 must be negative and so det(A) 7^ 0. Consider 
now the characteristic polynomial of A, Pa(X) = — X 3 + tr(A)X 2 — M 2 (A)X + det(A), where 
M 2 (A) denotes the sum of the principal 2x2 minors of A. Using the usual expression for 
the coefficients of Pa{X) in terms of eigenvalues of A, we find that M 2 (A) = Ai(— Ai) + 
A1A3 + (— Ai)A3 = |Ai| 2 > 0. There thus exists a 2 x 2 principal submatrix of A with positive 
determinant and negative trace. This submatrix is stable. □ 

We also have relationships between the stability of a matrix and its principal submatri- 
ces. In the case of symmetric matrices, it follows from Cauchy's interlacing theorem that all 
principal submatrices of a stable matrix are stable. 

Theorem 3.13 (Cauchy's interlacing theorem). Let A be a symmetric n x n matrix, and let 
B be an m x m principal submatrix of A, where m < n. If the eigenvalues of A are ot\ < 
...ctj... < a n , and those of B are (3\ < ...(3j... < (3 m , then for all j we have aj < (3j < a n - m+ j. 

Corollary 3.14. Any principal submatrix of a stable symmetric matrix is stable. Any sym- 
metric matrix containing an unstable principal submatrix is unstable. 

Even in the case of non-symmetric matrices, there are still some constraints of this type. For 
example, 

Lemma 3.15. Let A be an n x n matrix with strictly negative diagonal entries and n > 2. // 
A is stable, then there exists a 2 x 2 principal submatrix of A that is also stable. 

Proof. We use the formula for the characteristic polynomial in terms of sums of principal 
minors: 

P A (x) = (-l) n x n + (-lr^M^x™- 1 + (-l) n ~ 2 M 2 (A)x n ~ 2 + ... + M n (A), 

where Mk(A) is the sum of the k x k principal minors of A. (Note that M\ (A) = tr(A) 
and M n (A) = det(A).) The characteristic polynomial also has the well-known formula with 
coefficients that are symmetric polynomials in the eigenvalues; assuming A is stable, this yields 
M 2 (A) = Yli<j ^i^j > 0- This implies that at least one 2x2 principal minor is positive. Since 
the corresponding 2x2 principal submatrix has negative trace, it must be stable. □ 

In order to prove our main results in Section HJ we will also use the following well-known 
consequences of Cauchy's interlacing theorem. Here Ak refers to the principal submatrix 
obtained by taking the upper left k x k entries of A. 

Lemma 3.16 (Stable symmetric matrices). Let A be a real symmetric n x n matrix. Then 
A is stable iff (-l) k det(A k ) > for all 1 < k < n. 

Corollary 3.17. Let A be a real symmetric nxn matrix. Then A is stable iff (— 1)'°"' det(A a ) > 
for every principal submatrix A a . 
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4 Maximally flexible networks 



In this section we use the matrix results from Section [3] in order to prove our main results, 
Theorems 11.51 H-71 and 11.81 characterizing maximally flexible networks. 

4.1 Flexible vs. marginal cliques 

Recall that all flexible cliques are marginal cliques, because they can be made both stable 
and unstable via arbitrarily small perturbations of the network's connection strengths. The 
converse is not true. The following lemma gives simple, but incomplete, conditions for deter- 
mining whether or not a marginal clique is flexible in threshold-linear networks. 

Lemma 4.1. Let a be a marginal clique of a threshold-linear network (J,D)c- 

1. If there exists r e X(G) such that either (i) r C a and r unstable, or (ii) r ^) a and r 
stable, then a is not a flexible clique. 

2. If, on the other hand, (i) for all r C a, r is a stable clique, and (ii) for all r e X(G) 
such that r D a, t is an unstable clique, then a is a flexible clique. 

The proof follows from observing that any marginal clique can be perturbed to become stable 
or unstable by adding a multiple of the identity matrix to the corresponding principal subma- 
trix, and one can always find a small enough perturbation so that the stability of all stable and 
unstable principal submatrices in the original matrix is preserved. It is thus straightforward 
to check the flexibility of marginal cliques if certain patterns of stable/unstable cliques are 
also present. This is illustrated in the following example. 

Example 4.2. Consider the following matrices — D + J for (unconstrained) threshold-linear 
networks {D,J) G A/"(3): 

/ -1 -2 \ / -1 -1 1 \ / -1 2 1 \ 

Mi = -2 -1 , M 2 =1-1-1 , and M 3 = 1 -1 . 
\ -2 -1 / \ 1 -1 / \ -1 -1 / 

Mi. {1, 2, 3} is a flexible clique since it is marginal and all contained cliques are stable. 

M.2. {1, 2} is a marginal clique but it is not flexible, since {1, 2, 3} is stable. 

M 3 : {1, 2, 3} is a marginal clique but it is not flexible, since {1, 2} is unstable. 

Note that Lemma 14.11 says nothing about the situation where the cliques contained by or 
containing a given marginal clique are themselves also marginal. It is much more difficult to 
check for flexible cliques in a network with many marginal cliques. We investigate precisely 
this case, as we look for properties of networks with the maximal number of flexible cliques. 
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4.2 Proof of Theorem 11.51 



We begin with an example of a network in which all the cliques with at least two neurons 
are flexible. Such a network is maximally flexible, and provides a reference point in proving 
that all rank 1 networks are maximally flexible in J\f(n). The proof relies on the following 
determinant formulas: 

Lemma 4.3. Let W n (e,a), for n > 2, be the symmetric n x n matrix with entries 

{-1, if i=j, 
-1 + ae if {i,j} = {1,2}, (12) 
-1 + e if {i,j}^ {1,2}. 

Then, 

det W n (e, a) = (-l) n ae n ~ l (2n - 2 - (2n - 4)e - (n - 2 - (n - 3)e)a) . (13) 
In particular, 

det W n (e, 1) = (-l) n e n_1 (n - (n - l)e) . (14) 

Proof. This is a straightforward determinant computation. □ 

We now show that the matrix with all entries —1 corresponds to a network on n neurons 
that has the maximal number 2 n — n — 1 of flexible cliques, and is thus maximally flexible. 

Proposition 4.4. Let (J,D) G M{n) be the network with the matrix —D + J = —1, where 
— 1 is the n x n matrix having all entries —1. Then any subset o C {1, . . . ,n} with at least 
two neurons is a flexible clique. In particular, flex(J, D) — 2 n — n — 1. 

Proof. Let a be any subset with \o~\ = k > 2 neurons. To show that o is flexible, it suffices to 
show that there exists an e > so that for every < e < £q, there exist e-perturbations A s 
and A u of (J, D) under which a becomes a maximally stable clique and a minimally unstable 
clique, respectively. We show this via explicit construction of A s and A u . 

Let A s be the symmetric matrix with entries on the diagonal, entries (A s )ij = e for 
distinct i,j G a, and (As)ij = ~ e if either i ^ a or j ^ a. Clearly, A s is an e-perturbation. We 
need to show that a is a maximally stable clique for (J + A s , D); i.e., a is a stable clique of the 
perturbed network, and any clique r which properly contains a is unstable. By Theorem 11.21 
it is enough to show that the corresponding principal sub matrices of — 1 + A s are stable and 
unstable, respectively. 

Recall (fl2l) . and note that the principal submatrix (—1 + A s ) a = Wk(s, 1), where k = \a\. 
Using (fT4|) . we obtain 

det(-l + A s ) a = {-lfe k -\k — (k — l)e). (15) 

Note that the same expression holds for any a' C a, with k = \o~'\. By Corollary 13.171 it 
follows that (—1 + A s ) a is stable for all < s < 1. To show that a is maximally stable, 
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observe that any clique r properly containing a must also contain an order 2 clique whose 
corresponding principal submatrix is 



-1 -l-e 
-1-e -1 

which is unstable for all e > 0. Since the matrix (—1 + A S ) T is symmetric, it follows from 
Corollary 13. 141 that (—1 + A S ) T , for any t D a, is unstable for all < e < 1. 

To generate a perturbation A u for which the clique a is minimally unstable, we proceed 
as follows. Let < e < 1, and choose two neurons ii,^ G cr such that ii = min(o") and 
%2 = min(<7 — {h})- Let A u be the symmetric matrix with entries (A u )ij = e for distinct 
i,j G a unless {i,j} = {hjh}- We let the entries (A u ) ili2 = (A u ) i2il = ae, with a to be 
determined later. All other entries of A u are set to 0. To show that a is minimally unstable, 
we need to choose a so that (—1 + A u ) a is unstable while all its proper principal submatrices 
are stable. Since —1 + A u is symmetric, Corollary 13.171 tells us that this is accomplished if 
the determinant of (—1 + A u ) a has the 'wrong' sign (— 1) +1 , where k = \a\, and all j x j 
principal minors of (—1 + A u ) a , with j < k, have the 'right' sign (— 

Observing that (—1 + A u ) a = Wk{e, a), we have from (fT3l that 

det(-l + A u ) a = (-l) k ae k - 1 (2k - 2 - (2k - 4)e - (k - 2 - (k - 3)e)a). (16) 

There are two types of proper principal submatrices. The first are those that correspond to 
the cliques r C a that contain both i\ and 22, with j = |r|, and are equal to the matrices 
Wj(e,a). From (fT3|) . these have determinants 

det(-l + A U ) T = ( li'n '(2/ - 2 - (2j - 4)e - (j - 2 - (j - 3)e)a). (17) 

The second type of principal submatrices correspond to cliques v C o that do not contain 
both ii and ^. Letting j = these are equal to the matrices Wj(e, 1), and by ffl^l) have 
determinants 

det(-l + A u ) v = ( 1 )•'."' ; (./ - (j - l)e). (18) 

Using Corollary 13.171 we see that the cliques of type v are all stable for < e < 1 . It remains 
to choose a so that (TlBl) has sign (— l) fc+1 and ffTT|) has sign (— 1) J for all j = 2, fc — 1. 
For k > 3, we choose a so that 

2A; - 2 - (2fc - 4)e . / 2j - 2 - (2j - 4)e | 
< a < mm < — : ^ > 

j=2,...,k-l 



k — 2 — (k — 3)e { j - 2 - (j - 3)e 



This is always possible, since for < e < 1 the sequence on the right is decreasing; the 
minimum is attained for j — k — 1 and is greater than the term on the left, corresponding to 
j = k. Since for k > 3 we also have ae < 2|5§ £ the matrix A u is a 4e-perturbation. When 
k — 3, one can choose < e < \ and a = 4, while in the case k = 2 one needs simply to 
choose a < so that a is a minimally unstable clique. □ 

We now show that all the symmetric rank 1 networks are maximally flexible: 

Proposition 4.5. Let (J,D) G Af(n) be a symmetric rank 1 network. Then (J,D) is maxi- 
mally flexible in M(n) and has flexibility 2 n — n — 1. 
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Proof. Recall that by definition (J, D) £ M{n) is a symmetric rank 1 network if the matrix 
—D + J is a symmetric rank 1 matrix. Since the matrix — D + J has negative entries on the 
diagonal, there exists a vector x £ M™ so that — D + J = —xx T . Let diag(x) be the n x n 
diagonal matrix associated to the vector x £ ~R n . Then 

— .D + J = —xx T = diag(x) (—1) diag(x), 

where —1 is the n x n matrix with all entries —1 that we encountered in Proposition 14.41 

Since multiplication of a matrix on the left and right by the same diagonal matrix does 
not alter the sign of any principal minor, we have for any perturbation A 

det(— xx T + dia,g{x)A diag(x)) cr = det(— 1 + A) a , 

for any a C {1, n}. Moreover, if A is a symmetric perturbation, then so is diag(x)A diag(x), 
and the stability of any principal submatrix of — 1+A or — xx T +diag(x) A diag(x) is determined 
entirely by the signs of the principal minors (Corollary 13.1 7\i . We can thus obtain stable and 
unstable perturbations A s and A u of —xx T for any subset a consisting of |<r| > 2 neurons by 
modifying the perturbations A s and A u in Proposition l4.4l accordingly: A s = diag(x) A, diag(x) 
and A u = dia,g(x)A u diag(x). We conclude that the network (J,D) is maximally flexible in 
J\f(n), with flexibility 2 n - n - 1. □ 

Before proving Theorem 11.51 which extends the above results to G-constrained nonsym- 
metric networks in Af(G), we need to define the notion of 'pruning' of a network. We say that 
a graph G is a pruning of the graph G if the two graphs have the same vertices and the edges 
of G form a subset of the edges of G. We say that a network (J, D)q is a pruning of (J, D)g if 
G is a pruning of G, D = D, and = for all edges (ij) £ G. The following lemma shows 
that flexible cliques are 'inherited' by pruning. 

Lemma 4.6. Let (J,D)q be a pruning of (J,D)g- Consider a clique a £ X(G) C X(G). If 
a is a flexible clique of (J, D)q, then a is also a flexible clique of (J \D)q. 

Proof. This follows from the Definition 11.31 of flexible cliques. If a £ X(G) is a flexible 
clique, then there exist perturbations A s and A u consistent with G so that o is maximally 
stable for (J + A s , D)g and minimally unstable for (J + A u , D)q. If we also have a £ X(G), 
define the perturbations A s and A u , consistent with G, by setting all the entries in A s and A u 
corresponding to the pruned edges to 0. Since X(G) C X(G), it follows that the perturbations 
A s and A u realize a as a flexible clique of ( J, D) G . □ 

We now have all the ingredients necessary for proving Theorem 11.51 

Proof of Theorem 11.51 We prove first that all rank 1 networks (J, D) £ M{n) are maxi- 
mally flexible. Since the matrix — D + J has rank 1 and negative entries on the diagonal, 
there exists two vectors x, y £ W 1 , with > for all % = 1, n, so that — D + J = —xy T . 

Using these two vectors, we construct the diagonal matrix d = diag («/—)■ Let P be the 



matrix obtained from — D + J by conjugation with the matrix d, i.e. P = d(—D + J)d 1 . It 
has entries 
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and is therefore a rank 1 symmetric matrix. By Proposition 14.51 it follows that the network 
(dJd' 1 ,dDd~ l ) is a maximally flexible network in M{n). Since P and —D + J are similar 
matrices, related via conjugation by a diagonal matrix, it follows that all corresponding prin- 
cipal submatrices P a and (-D + J) a are also similar. Hence, a perturbation A of the network 
(J, D) has exactly the same stable and unstable cliques as a perturbation dAd~ x of the network 
(dJd~ x ,dDd~ l ). Since (dJd~ 1 ,dDd~ l ) is maximally flexible, it follows that (J,D) is also a 
maximally flexible network in M{n). 

Now let (J,D)g G N(G) be a G-constrained network with a rank 1 completion. We can 
think of the graph G as a pruning of the complete graph K n on n vertices. Let (J, .D) G M{n) 
be a rank 1 completion of the network (J, -D)g- By the previous arguments, the network 
(J, .D) is maximally flexible in Af(n), and has flexibility 2 n — n — 1. In particular, any clique 
cr g X(G) with |<t| > 2 is a flexible clique of the network (J,D). By Lemma [4.61 a is also a 
flexible clique of (J, D)q. Since has |X(G)| — n — 1 cliques with more than two neurons, 

it follows that the flexibility of (J, D) is |Jf(Cr)| — n — 1, which is maximal. □ 

4.3 Proof of Theorems [177] and HT81 

First we prove our main result, Theorem 11.71 

Proof of Theorem [T7TI Let (J,D)g be a maximally flexible threshold-linear network in 
J\f(G). This means that all the cliques a G X(G) with at least two neurons are flexible. Since 
all flexible cliques are marginal cliques, Theorem 11.21 gives that the corresponding principal 
submatrices of (-D + J) a are all marginally stable. In particular, all 2 x 2 and 3x3 principal 
submatrices are marginally stable, and thus by Lemma 13.121 it follows that 

det(-D + J) a = 0, for all a G X[G) with \a\ = 2 or 3. 

Applying Lemma 13.61 to —D + J, it follows that for all (ij) G G, the entry Jjj ^ 0. Thus 
the network (J,D)g has no silent connections. By Lemma 13.111 the homology condition 
H^XiG); Z) = implies that H 1 (X(G); Z 2 ) = H\X(G); R) = 0, and then by Proposition EH 
it follows that the matrix — D + J has a rank 1 completion. At the level of networks, this 
translates to (J, D)q having a rank 1 completion. □ 

Theorem 11.81 states that for the set M{n) of unconstrained threshold-linear networks, the 
maximally flexible networks are exactly the rank 1 networks. The proof is a direct application 
of Theorems 11.71 and 11.51 

Proof of Theorem 11.81 (=>) This direction is a direct consequence of Theorem 11.71 Let 
(J, D) G M{n) be a maximally flexible network. Its graph is the complete graph K n , and thus 
the corresponding clique complex X(K n ) is contractible and satisfies Hi(X(K n ),Z) = 0. By 
Theorem 11.71 it follows that (J,D) is a rank 1 network. (<s=) This follows from first part of 
Theorem 11.51 □ 

We also give a second proof of Theorem 11.81 without appealing to the homological argu- 
ments used in the proof of Theorem 11.71 
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Proof of Theorem 11.81 without homology /cohomology. (=^) Suppose (J, D) E J\f(n) is 
a maximally flexible network. This means (J, D) must have flexibility 2 n — n — 1. In particular, 
all 2 x 2 and 3x3 principal submatrices of — -D + J must be marginally stable, and so by 
Lemma 13.121 all 2 x 2 and 3x3 principal minors must vanish. This, together with the fact 
that the diagonal entries are strictly negative, implies that — D + J satisfies the hypotheses 
of Lemma f3.7[ whose proof does not rely on cohomology arguments, and is thus rank 1. (<=) 
This follows from first part of Theorem 11.51 which does not use homology or cohomology 
arguments. □ 
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